Skip to content

feat(ios): support external xctest runner artifact#806

Open
hannojg wants to merge 2 commits into
callstack:mainfrom
hannojg:hannojg/external-ios-xctest-runner
Open

feat(ios): support external xctest runner artifact#806
hannojg wants to merge 2 commits into
callstack:mainfrom
hannojg:hannojg/external-ios-xctest-runner

Conversation

@hannojg

@hannojg hannojg commented Jun 15, 2026

Copy link
Copy Markdown

Summary

agent-device might be used in contexts where you don't control the signing for iOS apps. One example is using agent-device for automation on device farm providers, such as AWS devicefarm.
We can't build the XCUITest package and sign it on the AWS devicefarm runner as no provisioning profiles are available there.
What works though, is building the XCUITest package beforehand without signing, upload it to AWS, and AWS will sign it for you so that it can be consumed.

This PR adds support to allow providing an "external" XCUITest package.

We already landed a similar approach in harness (for the same reason):

Test plan

AI was so kind to add some unit tests.
Additionally i tested those changes for myself on AWS and it works well. Let me know if anything else is needed :)

Copy link
Copy Markdown
Member

Took a careful pass (also in the context of #807, which this pairs with). The external-artifact feature is gated cleanly behind unset env vars, so the default path is unaffected. Two things I'd want to resolve before merge, plus a minor:

1. Session reuse treats every external session as stale. resolveReusableRunnerSession (runner-session.ts:265-284) computes expectedDerived = resolveRunnerDerivedPath(...) (the cache-convention path) and tears the session down if it doesn't equal existing.xctestrunArtifact.derived. For cache === 'external', derived is the external/configured path, which never matches → the runner is torn down and relaunched on every command. This wasn't updated alongside the markRunnerXctestrunArtifactBadForRun guard. It also defeats prewarm in external mode (the just-warmed session is discarded on the first real command). Suggest short-circuiting this check for external, mirroring the bad-artifact guard.

2. Relocating the env overlay may break __TESTROOT__. prepareXctestrunWithEnv now writes the overlay .xctestrun into AGENT_DEVICE_IOS_XCTEST_ENV_DIR, a different dir than the source. If the external xctestrun uses __TESTROOT__-relative TestBundlePath/TestHostPath (xcodebuild's default), __TESTROOT__ resolves relative to the overlay's location → wrong bundle paths. This presumably works because the AWS-produced xctestrun uses absolute paths — worth a comment documenting that assumption, or rewriting __TESTROOT__ to the original dir.

3. (minor) Default derived may be read-only. When AGENT_DEVICE_IOS_XCTEST_DERIVED_DATA_PATH is unset, derived defaults to path.dirname(xctestrunPath) — on AWS that's the read-only artifact mount, now passed as -derivedDataPath where xcodebuild writes logs/results. The override exists, but the default invites a confusing failure; consider defaulting to a temp dir.

Nice test coverage on the new functions otherwise. (1) is the one I'd really like confirmed given you're targeting device farms + prewarm.


Generated by Claude Code

@thymikee

thymikee commented Jun 15, 2026

Copy link
Copy Markdown
Member

btw, instead of using env vars we could reuse our config mechanism and add new keys (e.g. iosXctestrunFile, iosXctestDerivedDataPath, iosXctestEnvDir). We'll keep the env-var ergonomics device farms want (the daemon reads them in-process), and also get config-file support, a typed/validated surface, auto-generated AGENT_DEVICE_* names, and docs, and it stays consistent with how every other setting in the tool is plumbed.

#807 can stay as an env var because it realistically can't be anything else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants